• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    METHOD, SYSTEM AND STORAGE MEANS OR LOGIC TO MIX CLOUDS OF INFRARED AND COLOR COMPONENT DATA POINTS. The present invention relates to mixing RGB data with infrared data in order to provide relative depth data in regions where infrared data are sparse. Infrared data, such as corresponding to point cloud data, are processed to determine sparse regions therein. For any such sparse regions, the RGB data that corresponds to a counterpart region in the RGB data is added to a data structure. The data structure, which can include or be concatenated with the IR data, can be used for depth-related data, for example with a point cloud.
    公开号:BR112015025974B1
    申请号:R112015025974-0
    申请日:2014-04-14
    公开日:2022-01-25
    发明作者:Patrick John Sweeney；David F. Harnett
    申请人:Microsoft Technology Licensing, Llc；
    IPC主号:
    专利说明:

    BACKGROUND
    [0001] In active depth detection detection, a projector projects light patterns such as infrared (IR) dots to illuminate a region being detected. Projected patterns are captured by a camera/sensor (two or more in stereo systems), with the image (or images) processed to compute a depth map or the like, eg per frame. Infrared is advantageous because color (RGB) images result in very noisy depth values.
    [0002] In stereo systems, stereo cameras capture two images from different points of view. So, for example, one way to perform a depth estimate with a pair of stereo images is to find matches between the images, for example, to correlate each projected and detected IV point in one image with a counterpart IV point in the other image. Once matched, the patterns projected within the images can be correlated with each other, and the disparities between one or more features of the correlated points used to estimate a depth for that specific pair of points. For example, a dense depth map at the original (native) camera resolution can be obtained by area matching (eg through a 5x5 size window).
    [0003] However, not all surfaces reflect IR light specifically well. As a result, in any part of an image that corresponds to a weakly reflective IR surface, there is usually not enough IR data (e.g. reflected points) in the stereo images to correlate one with the other, and thus no depth data. or very sparse depth data. This is problematic even with a single two-dimensional depth map; in point cloud applications, such as those that use depth data to build a mesh, the lack of adequate depth data in certain regions can be even more pronounced. SUMMARY
    [0004] This Table of Contents is provided to introduce a selection of representative concepts in a simplified form which are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way that would limit the scope of the claimed subject.
    [0005] Briefly, one or more of several aspects of the subject described here are directed towards using color component data (eg RGB) to augment infrared data in regions where infrared data is sparse. One or more aspects are directed at getting IR data that have IR data points and color component data that have color component data points that correspond to an image plane. When determining a region in the IR image data that is a sparse region with respect to having IR data points in it, at least some color component data that correspond to color component data points of a counterpart region in the color component image data are added to a data structure.
    [0006] In one or more aspects, an image processing component is configured to receive infrared images captured by at least one infrared (IR) camera, and counterpart red, green, blue (RGB) images captured by at least one least one RGB camera. A mixing component coupled to or incorporated into the image processing component processes infrared data corresponding to an infrared image to determine one or more sparse regions therein. For each sparse region, the Blend component adds RGB data that corresponds to a counterpart region to a counterpart RGB image to the data structure.
    [0007] One or more aspects are directed to obtain first IR data that correspond to point cloud data projected over a foreground image and to obtain first RGB data that correspond to point cloud data projected over the first image plane. One or more sparse infrared regions in the first IR data and one or more counterpart regions of the sparse IR regions in the first RGB data are determined. At least some RGB data from one or more counterpart regions is added to a data structure. This can be performed with a second set of IR data and RGB data, and so on, to add RGB data to the data structure in regions where the IR Data is sparse.
    [0008] Other advantages may become apparent from the following detailed description when taken in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
    [0009] The present invention is illustrated by way of example and not limited in the accompanying Figures in which like reference numerals indicate similar elements and in which:
    [0010] Figure 1 is a block diagram representing exemplary components that can be used to augment sparse infrared data with RGB data, according to one or more exemplary implementations.
    [0011] Figure 2 is a representation of an example of projecting points within a scene, to obtain IR data according to one or more exemplary implementations.
    [0012] Figure 3 is a representation of how the projected points can be used in a multi-camera set scenario, according to one or more exemplary implementations.
    [0013] Figure 4 is a representation of how sparse detected infrared data can be in an area that reflects IR light weakly, according to one or more exemplary implementations.
    [0014] Figure 5 is a representation of how sparse infrared regions can be determined from IR data, according to one or more exemplary implementations.
    [0015] Figures 6 is a representation of how sparse regions can be varied, according to one or more exemplary implementations.
    [0016] Figure 7 is a flowchart depicting exemplary steps that can be taken to determine sparse IV regions and add counterpart RGB region data to a data structure, such as for use as depth-related data, in accordance with with one or more exemplary implementations.
    [0017] Figure 8 is a block diagram representing an exemplary non-limiting computing system or operating environment in which one or more aspects of the various embodiments described herein may be implemented. DETAILED DESCRIPTION
    [0018] Various aspects of the technology described here are generally aimed at using RGB data to estimate depth data in regions where infrared data are sparse. In one aspect, the density of IR data (eg points projected from an IR point cloud) is evaluated in several subparts so as to augment any low density (IR-based) subpart with RGB data. In this way, mixing the RGB and IR stereo points based on density provides a point cloud with more complete depth data.
    [0019] In one aspect, density is computed over multiple subparts of a two-dimensional image plane that corresponds to a projection of the infrared point cloud over the image plane. Subparts can be arranged as a grid or similar. For any grid cell, the IR point density is evaluated, and if too low, the RGB point data for that grid cell is kept, otherwise the RGB point data is discarded. The process can be repeated for as many sets of cameras (capsules) that were used to capture the point cloud data in a given configuration.
    [0020] It should be understood that any of the examples here are not limiting. For example, although a three-dimensional point cloud captured by multiple arrayed capsules is exemplified here, the technology described herein can be applied to as little as a single two-dimensional depth map. Also, although RGB (red, green, blue) color component data is described, the RGB sensor can be replaced by or augmented with an ultraviolet sensor to fill in the sparse IR data. As such, the present invention is not limited to any specific embodiments, aspects, concepts, structures, functionalities, or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionality, or examples described herein are non-limiting, and the present invention may be used in various ways that provide benefits and advantages in active depth detection, point clouds, and image processing. generally.
    [0021] Figure 1 shows an exemplary system in which a capsule 100 comprising stereo IR cameras 101 and 102, stereo RGB cameras 103 and 104, and a projector 106 (e.g., an IR laser diffracted into many thousands of dots) captures IR images and stereo RGB images. Note that the capsule 100 is only an exemplary arrangement, and that in other arrangements, the cameras 101-104 may be arranged in any order relative to each other. Indeed, in one implementation the projector is positioned above the cameras. Also, any of the cameras and/or the projector can be separate from each other, rather than being part of any capsule configuration; no capsule is needed. Thus, Figure 1 is only showing components for purposes of explanation, and without scaling, relative dimensions, relative positions, combination of devices within a capsule housing/device and so on should be inferred from Figure 1.
    [0022] In the example of Figure 1, the capsule 100 is coupled to (or combined with) an image capture system or subsystem 108. The cameras are generally controlled, for example, through a camera interface 110 and controller 111, to capture time-synchronized stereo images (eg cameras are "genlocked"). In one implementation, cameras 101 and 102 capture infrared (IR) images 114, as IR is highly effective in estimating depth in varying light conditions and does not affect the visible appearance of the scene. Also, cameras 103 and 104 capture the 115 stereo RGB images. As can be readily appreciated and as exemplified below, in some scenarios such as studio environments, more than one such capsule and image capture system/subsystem may be present. gifts.
    [0023] In Figure 1, a projector 106 is shown that projects an IR pattern over a scene, such as a dot pattern (e.g. dots) or a line pattern, although other dot shapes and/or or pattern types can be used. For purposes of brevity, points are generally described hereinafter. Illuminating the scene with a relatively large number of points of distributed infrared, the IR cameras 102 and 103 capture texture data as part of the infrared image data. Note that projector 106 is shown as coupled to controller 112 via projector interface 116; any such control can be as simple as turning the projector on and off or using power saving modes, however more complex control such as pulsing, changing dot distribution, changing intensity and/or the like is feasible.
    [0024] Figure 2 exemplifies this projection concept. Projector 106, represented as a circle between stereo cameras 101 - 104, projects a pattern of dots over a scene 222. Cameras 101 - 104 capture the dots as they reflect off object surfaces in scene 222 and (possibly) the bottom. In general, the disparity in the images that two cameras see is indicative of the distance to the reflective surface. Note that Figure 2 is not intended to be to scale, nor does it convey any sizes, distance, dot distribution pattern, dot density, and so on.
    [0025] Returning to Figure 1, the images captured by cameras 101 - 104 are provided to an image processing system or subsystem 118. In some implementations, the image processing system 118 and image capturing system or subsystem 104, or its parts, can be combined into a single device. For example, a home entertainment device might include all of the components shown in Figure 1 (as well as others not shown). In other implementations, parts (or all) of the image capture system or subsystem 104, such as the cameras and projector, may be a separate device that docks to a game console, personal computer, mobile device, dedicated processing device, and /or similar.
    [0026] The image processing system or subsystem 118 includes a processor 120 and a memory 122 that contains one or more image processing algorithms, including an IR/RGB mixing algorithm 124 as described herein. In general, the IR/RGB mixing algorithm 124 outputs a list 130 or other suitable data structure that includes IR points and RGB points that have associated values from which the depth can be determined. Also shown in Figure 1 is an interface 132 to the image processing system or subsystem 118, such as for connecting a keyboard, game controller, display, pointing device, microphone for voice commands and/or the like as appropriate for a user interact with an application or similar that uses the IR/RGB mixed point list 130.
    [0027] Figure 3 shows a plurality of capsules 3001 - 3004 arranged to capture images of an object (eg a person) from different perspectives. Note that although four such capsules are shown in Figure 3, it is understood that any practical number may be present in a given configuration. For example, such a configuration uses nine pods, with two sets of four pods at different heights encircling a space plus a pod above the space.
    [0028] In the example of Figure 3, the IR and RGB image data captured from each of the four (or more) capsules form separate IR and RGB point clouds, (IV-PC and RGB-PC, respectively). The point cloud data is made available to the IR/RGB point cloud mixing algorithm 124 for processing as described herein.
    [0029] As generally depicted in Figure 3, the projectors of each capsule project the pattern of light (IR dots) onto an object, eg a person 330. Reflected IR light is captured in each capsule 3001 - 3004 However, as exemplified in Figure 3, (and also in enlarged form in Figure 4), some surfaces (which correspond to area 332) do not reflect IR light well. Human hair is an example, however many materials, fabrics and the like have poor IR reflective properties. Note that the points in Figures 3 and 4 are for illustrative purposes only, and that the distribution, total density, sizes and so on are not intended to convey any actual sizes, density and/or distribution information.
    [0030] Thus, there are sparse IR data in area 332. As described here, this scarcity level is detected where it exists in the image, and RGB points added to regions where the IR points are sparse.
    [0031] To this end, given a point cloud, the point cloud data can be projected onto a two-dimensional image plane from any given perspective. Thus, one image plane for each capsule can have the point cloud data projected onto it, one image plane for the IR data, and the other for the RGB data. Alternatively (for example, if no point clouds exist), each frame of IR data and RGB data is captured as two-dimensional (2D) images in the cameras, whereby the image plane data exists from the capture.
    [0032] In any case, Figure 4 shows how a 2D 440 image can be separated into sub-parts as a grid to determine the density of reflected IR points captured therein. As can be seen from this simplified example, the density is low in the area 332 which corresponds to the surface that reflected IR light weakly. Note that the size of the grids and the number of grids are not intended to drive any actual size, relative size, relative number and so on.
    [0033] Virtually any technique for locating scarcity within subsections of a network can be used, and need not be grid-based, but grids provide a straightforward solution. For example, a direct way to detect scarcity uses a counting mechanism that counts the points in grid cells. For example, if the number of points in a given grid cell is below a threshold value, then that particular grid cell is considered sparse. In such a situation, the RGB data for this grid is kept to augment the sparse IV data, for use as desired (eg in depth detection).
    [0034] Figure 5 shows essentially the same example as Figure 4, except that the grid cells are unglued based on maximizing scarcity (or conversely maximizing density) or based on some other criterion. By shifting the grids (and/or lengthening or shortening them vertically and/or widening or narrowing them horizontally), different subparts are sparse. For example, in Figure 5, there are six grid cells (regions) near area 332 that are likely to be determined to be sparse, versus four such grid cells in the example of Figure 4. Note that adding RGB adds some noise to the depth detection , but in many cases having depth data with some noise in a region is an improvement over having no or only very sparse depth data in that region.
    [0035] Figure 6 shows another alternative, in which the grid cells can be arbitrarily sized and/or arranged, in order to increase or decrease the number of RGB points added. For example, what was a single grid cell 444 in Figure 4 is split into two smaller grid cells 662 and 664 in Figure 6. This is also represented by the solid lines around these cells, as are other smaller cells in Figure 6 Note that this can be done while counting, eg a cell not sparse but "just above the threshold" can be subdivided into subregions to determine if a part is now sparse, eg relative to a reduced threshold, such as half of the boundary if the grid cell area is divided in half. This tends to increase the RGB data added because sometimes what was a non-sparse cell can be split into a sparse part and a non-sparse part. Unlike to reduce the amount of RGB data added, cells sparse but "just below the threshold" can be subdivided to determine if a part is now non-sparse. Both of these alternatives can be done at the same time.
    [0036] Figure 7 is a flowchart representing exemplary steps of a suitable IR - RGB mixing algorithm. In general, the algorithm is repeated per capsule (through the 702 and 718) to obtain a comprehensive list of IR points augmented with RGB points in regions where the IR is sparse (as detected with respect to any capsule).
    [0037] Step 702 selects a capsule, and steps 704 and 706 obtain the 2D IR and RGB image planes for that capsule, for example via point cloud projection (or as captured by that capsule; note that for density determination purposes, only one of the stereo images can be used). Step 708 divides each image into subparts, for example, grids. Grids can be of any suitable size (eg sixteen by sixteen blocks).
    [0038] Step 710 selects an IR image plane grid, and counts the IR points in it. Step 712 determines whether the grid density is low, for example, by evaluating the count against a threshold. The threshold may be a fixed number (e.g., ten) that is reasonable, estimated, or computed for a given capsule configuration, projection pattern density, and distances to the illuminated object, or may be varied based on technique/analysis. statistics such as mean and standard deviation. The grid size and/or shapes (eg squares, rectangles, triangles, hexagons) and so on can likewise be varied.
    [0039] In either case, when a sparse grid cell or the like is detected in step 712, step 714 adds the counterpart grid cell RGB points to the RGB image plane data in a list or other grid structure. adequate data. If not sparse, then enough IV data is present in that region for use as desired.
    [0040] Step 716 repeats the process for the next grid, and so on, until all grids have been processed. Step 718 repeats the process for the other capsules until all capsules have been processed.
    [0041] When the capsules were processed, the RGB image data is in the list or similar. Duplicates may exist in the list, which can be removed via step 720; (note that alternatively, duplicates may not be added in step 714). Step 722 adds the IV points (from all capsules) to the list, and the process ends. The list now contains all IR points, plus RGB points where IR data was sparse for any given capsule.
    [0042] Note that in alternative implementations, the amount of scarcity may be a factor in adding RGB. For example, the closer the IV count is to the threshold then the less the number of RGB points that can be added. Similarly, the proximity of an RGB point to an IR point in the coordinate space can determine whether to add this point.
    [0043] As can be seen, a technology is described by which an image or point cloud with sparse IR data regions can be mixed with RGB data in the sparse regions. The resulting scrambled data can be used to obtain depth values where or only sparse IR data regions exist. EXEMPLARY OPERATING ENVIRONMENT
    [0044] Figure 8 illustrates an example of a suitable computing and network environment 800 in which computer-related examples and implementations described herein can be implemented, for example. The 800 computing system environment is only one example of a suitable computing environment and is not intended to suggest any limitations as to the scope of use or functionality of the invention. Nor should the computing environment 800 be interpreted as having any dependency or requirement relating to one or combination of components illustrated in the exemplary operating environment 800.
    [0045] The invention is operational with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems , microprocessor-based systems, set-top boxes, programmable consumer electronics, networked PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
    [0046] The invention can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so on, which perform specific tasks or implement specific abstract data types. The invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected via a communications network. In a distributed computing environment, program modules may be located on a local and/or remote computer storage medium including memory storage devices.
    [0047] Referring to Figure 8, an exemplary system for implementing various aspects of the invention may include a general purpose computing device in the form of a computer 810. The components of the computer 810 may include, but are not limited to, a unit 820, a system memory 830, and a system bus 821 that couples various system components including system memory into the processing unit 820. The system bus 821 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronic Standards Association (VESA) local bus , and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
    [0048] The 810 computer typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by the computer 810 and includes volatile and non-volatile media, and removable and non-removable media. By way of example, and not limitation, the computer readable medium may comprise a computer storage medium and a communication medium. The computer storage medium includes a volatile and non-volatile, and removable and non-removable medium implemented in any method or technology for storing information such as computer readable instructions, data structure, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile discs (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium which can be used to store desired information and which can be accessed by the computer 810. The communication medium typically incorporates computer-readable instructions, data structure , program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any means of providing information. The term "modulated data signal" means a signal that has one or more of its characteristics adjusted or changed in such a way as to encode the information in the signal. By way of example, and not limitation, the communication medium includes a wired medium such as a wired network or direct wired connection, and wireless medium such as an acoustic, RF, infrared, and other wireless medium. Combinations of any of the above may also be included within the scope of computer readable media.
    [0049] System memory 830 includes computer storage media in the form of volatile and/or non-volatile memory such as read-only memory (ROM) 831 and random access memory (RAM) 832. An input system /basic output 833 (BIOS), which contains the basic routines that help transfer information between elements within the computer 810, such as during startup, is typically stored in ROM 831. RAM 832 typically contains data and/or modules that are immediately accessible to and/or currently being operated by the processing unit 820. By way of example, and not limitation, Figure 8 illustrates the operating system 834, application programs 835, other program modules 836, and program data. 837.
    [0050] Computer 810 may also include other removable/non-removable, volatile/non-volatile computer storage media. As an example only, Figure 8 illustrates a hard disk drive 841 that reads from or writes to non-removable, non-volatile magnetic media, a magnetic disk drive 851 that reads from or writes to a removable, non-volatile magnetic disk 852, and an optical disk drive 855 that reads from or writes to a removable, non-volatile optical disk 856 such as a CD ROM or other optical medium. Other removable/non-removable, volatile/non-volatile computer storage media that may be used in the operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile discs, digital video tape, Solid State RAM, Solid State ROM, and the like. Hard disk drive 841 is typically connected to system bus 821 through a non-removable memory interface such as interface 840, and magnetic disk drive 851 and optical disk drive 855 are typically connected to system bus 821 by a removable memory interface, such as the 850 interface.
    [0051] The units, and their associated computer storage media, described above and illustrated in Figure 8, provide the storage of computer-readable instructions, data structures, program modules, and other data for the computer 810. In Figure 8 , for example, hard disk drive 841 is illustrated as storing operating system 844, application programs 845, other program modules 846, and program data 847. Note that these components may be either the same as or different from the system. operating system 834, application programs 835, other program modules 836, and program data 837. Operating system 844, application programs 845, other program modules 846, and program data 847 are given different numbers here to illustrate that at the very least, these are different copies. A user can enter commands and information into the computer 810 through input devices such as a tablet, or electronic digitizer, 864, a microphone 863, a keyboard 862, and a pointing device 861, commonly referred to as a mouse, trackball, or dashboard. Touch. Other input devices not shown in Figure 8 may include a joystick, gamepad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 820 through a user input interface 860 that is coupled to the system bus, but may be connected through other interfaces and bus structures, such as a parallel port, a game port or a universal serial bus (USB). A monitor 891 or other type of display device is also connected to the system bus 821 through an interface, such as a video interface 890. The monitor 891 may also be integrated with a touch screen panel or the like. Note that the display and/or touch screen panel may be physically coupled to a housing in which the computing device 810 is embedded, such as in a tablet-type personal computer. In addition, computers such as computing device 810 may also include other peripheral output devices such as speakers 895 and printer 896, which may be connected through peripheral output interface 894 or the like.
    [0052] The 810 computer can operate in a network environment using logical connections to one or more remote computers, such as a remote 880 computer. The 880 remote computer can be a personal computer, a server, a router, a network PC , a point device or other common network node, and typically includes many or all of the elements described above in relation to the computer 810, although only one memory storage device 881 is illustrated in Figure 8. The logical connections shown in Figure 8 includes one or more local area networks (LAN) 871 and one or more wide area networks (WAN) 873, but may also include other networks. Such network environments are common in offices, enterprise-wide computer networks, intranets, and the Internet.
    [0053] When used in a LAN network environment, the 810 computer is connected to the 871 LAN through an 870 network interface or adapter. When used in a WAN network environment, the 810 computer typically includes an 872 modem or other means to establish communications over WAN 873, such as the Internet. Modem 872, which may be internal or external, may be connected to system bus 821 through user input interface 860 or other appropriate mechanism. A wireless network component 874 such as comprising an interface and antenna may be coupled through a suitable device such as an access point or point computer to a WAN or LAN. In a network environment, program modules presented in connection with computer 810, or portions thereof, may be stored on the remote memory storage device. By way of example, and not limitation, Figure 8 illustrates remote application programs 885 as residing in memory device 881. It can be appreciated that the network connections shown are examples and other means of establishing a communications connection between computers may be used. .
    [0054] An auxiliary subsystem 899 (e.g. for displaying auxiliary content) may be connected through the user interface 860 to allow data such as program content, system status and event notifications to be provided to the user even if the main portion of the computer system is in a low power state. Auxiliary subsystem 899 may be connected to modem 872 and/or network interface 870 to allow communication between these systems while main processing unit 820 is in a low power state.
    [0055] Alternatively, or in addition, the functionality described here may be performed, at least in part, by one or more logical hardware components. For example, and without limitation, illustrative types of hardware logic components that may be used include Field Programmable Gate Networks (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System Systems on Chips (SOCs), Complex Programmable Logic Devices (CPLDs), etc. CONCLUSION
    [0056] Although the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. It should be understood, however, that there is no intention to limit the invention to the specific forms described, but rather, the intention is to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the invention.
    权利要求:
    Claims (10)
    [0001]
    1. Method for determining depth, characterized in that it comprises obtaining (704) stereo infrared image data corresponding to an infrared image plane with infrared data points, obtaining (706) color stereo image data corresponding to the image plane with image color data points, determine (712) a region in the infrared image data that is a sparse region with respect to having infrared data points in it, and add (714) at least some color data corresponding to the color data points of a counterpart region in the color component image data to a data structure that includes the infrared image data points and the image color data points having associated values and determine the depth of the data structure.
    [0002]
    2. Method according to claim 1, characterized in that obtaining (704) the infrared image plane comprises projecting infrared data from a point cloud.
    [0003]
    A method as claimed in claim 1, further comprising dividing (708) the image plane into a plurality of regions, including a region comprising the sparse region.
    [0004]
    A method according to claim 1, characterized in that adding (714) the at least some color component data comprises adding an amount of color component data based on a scarcity level of the region.
    [0005]
    A method as claimed in claim 1, further comprising dividing (708) the image plane into a plurality of regions, including a region comprising the sparse region, in which a size of at least some of the regions are determined based on density data.
    [0006]
    6. Method according to claim 1, characterized in that it further comprises using the infrared data points and the color component data points in the data structure to determine the depth data.
    [0007]
    7. System, characterized in that it comprises an image processing component (118), the image processing component configured to receive stereo infrared images (114) captured by stereo infrared cameras (101, 102) and equivalent stereo RGB images (115) captured by stereo RGB cameras (103, 104) and a mixing component (124) coupled to or incorporated into the image processing component, the mixing component configured to process infrared data corresponding to an infrared image to determine a or more sparse regions in the same and for each sparse region, to add RGB data corresponding to an equivalent region in an RGB image equivalent to a given structure (130) that includes the infrared data and the RGB data with associated values and to determine the depth of the data structure.
    [0008]
    8. System according to claim 7, characterized in that the image processing component (118) is configured to receive infrared image data corresponding to infrared point cloud data and RGB data that correspond to RGB point cloud data that is based on images from a plurality of stereo camera arrays.
    [0009]
    9. System according to claim 7, characterized in that the mixing component (124) is configured to add RGB data from at least two different sets of RGB data obtained from two different sets of stereo cameras.
    [0010]
    10. Computer readable storage medium, characterized in that it has a method as defined in any one of claims 1-6.
    类似技术:
    公开号 | 公开日 | 专利标题
    BR112015025974B1|2022-01-25|Method for determining the depth and computer-readable storage medium
    Cabral et al.2014|Piecewise planar and compact floorplan reconstruction from images
    Kutulakos et al.1999|A theory of shape by space carving
    KR101310589B1|2013-09-23|Techniques for rapid stereo reconstruction from images
    US9460553B2|2016-10-04|Point-based global illumination directional importance mapping
    MacDonald et al.2010|Polynomial texture mapping and 3d representations
    BR102013028714A2|2014-10-07|FILL POINTS IN A CLOUD OF POINTS
    Kersten et al.2012|Potential of automatic 3D object reconstruction from multiple images for applications in architecture, cultural heritage and archaeology
    Wang et al.2005|Camera calibration and 3D reconstruction from a single view based on scene constraints
    US11189043B2|2021-11-30|Image reconstruction for virtual 3D
    WO2018077071A1|2018-05-03|Panoramic image generating method and apparatus
    EP2856431A2|2015-04-08|Combining narrow-baseline and wide-baseline stereo for three-dimensional modeling
    JP2004220312A|2004-08-05|Multi-viewpoint camera system
    KR20190112894A|2019-10-08|Method and apparatus for 3d rendering
    Ding et al.2009|Multiperspective stereo matching and volumetric reconstruction
    US10008025B2|2018-06-26|Rendering system and rendering method thereof
    US8872824B1|2014-10-28|System, method, and computer program product for performing shadowing utilizing shadow maps and ray tracing
    KR20190068168A|2019-06-18|Apparatus for generating 3-dimensional image and method for the same
    Peethambaran et al.2017|Enhancing Urban Façades via LiDAR‐Based Sculpting
    JP6259262B2|2018-01-10|Image processing apparatus and image processing method
    Ni et al.2014|3D reconstruction of small plant from multiple views
    Nguyen et al.2013|A hybrid image-based modelling algorithm
    Kramer et al.2012|Multiple kinects
    Zhang et al.2010|Photorealistic 3D Volumetric Model Reconstruction by Voxel Coloring
    Skuratovskyi et al.2019|Outdoor Mapping Framework: From Images to 3D Model
    同族专利:
    公开号 | 公开日
    BR112015025974A8|2020-01-14|
    KR20150143751A|2015-12-23|
    AU2014254218A1|2015-10-29|
    RU2660596C2|2018-07-06|
    MX2015014570A|2016-06-29|
    KR102171231B1|2020-10-28|
    US20140307056A1|2014-10-16|
    BR112015025974A2|2017-07-25|
    RU2015143935A|2017-04-19|
    JP2016522923A|2016-08-04|
    CN105706143B|2019-02-22|
    EP2987139A1|2016-02-24|
    MX352449B|2017-11-24|
    JP6562900B2|2019-08-21|
    CA2908689A1|2014-10-23|
    US9191643B2|2015-11-17|
    EP2987140B1|2018-12-05|
    WO2014172230A1|2014-10-23|
    AU2014254218A8|2016-01-14|
    CN105706143A|2016-06-22|
    AU2014254218B2|2017-05-25|
    US20140307952A1|2014-10-16|
    CA2908689C|2021-08-24|
    CN105229697A|2016-01-06|
    EP2987140A1|2016-02-24|
    US20190379873A1|2019-12-12|
    WO2014172226A1|2014-10-23|
    CN105229697B|2019-01-22|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    US5812787A|1995-06-30|1998-09-22|Intel Corporation|Video coding scheme with foreground/background separation|
    US7606417B2|2004-08-16|2009-10-20|Fotonation Vision Limited|Foreground/background segmentation in digital images with differential exposure calculations|
    US8330831B2|2003-08-05|2012-12-11|DigitalOptics Corporation Europe Limited|Method of gathering visual meta data using a reference image|
    US20030063185A1|2001-09-28|2003-04-03|Bell Cynthia S.|Three-dimensional imaging with complementary color filter arrays|
    US7227893B1|2002-08-22|2007-06-05|Xlabs Holdings, Llc|Application-specific object-based segmentation and recognition system|
    US20040169873A1|2003-02-28|2004-09-02|Xerox Corporation|Automatic determination of custom parameters based on scanned image data|
    GB0308943D0|2003-04-17|2003-05-28|Univ Dundee|A system for determining the body pose of a person from images|
    GB0326374D0|2003-11-12|2003-12-17|British Telecomm|Object detection in images|
    US7657126B2|2005-05-09|2010-02-02|Like.Com|System and method for search portions of objects in images and features thereof|
    US7676081B2|2005-06-17|2010-03-09|Microsoft Corporation|Image segmentation of foreground from background layers|
    US7756325B2|2005-06-20|2010-07-13|University Of Basel|Estimating 3D shape and texture of a 3D object based on a 2D image of the 3D object|
    US7680314B2|2005-10-17|2010-03-16|Siemens Medical Solutions Usa, Inc.|Devices, systems, and methods for improving image consistency|
    US8848053B2|2006-03-28|2014-09-30|Objectvideo, Inc.|Automatic extraction of secondary video streams|
    US7885463B2|2006-03-30|2011-02-08|Microsoft Corp.|Image segmentation using spatial-color Gaussian mixture models|
    EP1862969A1|2006-06-02|2007-12-05|Eidgenössische Technische Hochschule Zürich|Method and system for generating a representation of a dynamically changing 3D scene|
    KR100829581B1|2006-11-28|2008-05-14|삼성전자주식회사|Image processing method, medium and apparatus|
    JP2009199284A|2008-02-21|2009-09-03|Univ Of Tokyo|Road object recognition method|
    US8249349B2|2008-11-25|2012-08-21|Microsoft Corporation|Labeling image elements|
    US8681216B2|2009-03-12|2014-03-25|Hewlett-Packard Development Company, L.P.|Depth-sensing camera system|
    RU2421933C2|2009-03-24|2011-06-20|Корпорация "САМСУНГ ЭЛЕКТРОНИКС Ко., Лтд."|System and method to generate and reproduce 3d video image|
    US20100293179A1|2009-05-14|2010-11-18|Microsoft Corporation|Identifying synonyms of entities using web search|
    JP5322789B2|2009-06-15|2013-10-23|三菱電機株式会社|Model generation apparatus, model generation method, model generation program, point cloud image generation method, and point cloud image generation program|
    US8537200B2|2009-10-23|2013-09-17|Qualcomm Incorporated|Depth map generation techniques for conversion of 2D video data to 3D video data|
    US8355565B1|2009-10-29|2013-01-15|Hewlett-Packard Development Company, L.P.|Producing high quality depth maps|
    RU2426172C1|2010-01-21|2011-08-10|Корпорация "САМСУНГ ЭЛЕКТРОНИКС Ко., Лтд."|Method and system for isolating foreground object image proceeding from colour and depth data|
    US8787663B2|2010-03-01|2014-07-22|Primesense Ltd.|Tracking body parts by combined color image and depth processing|
    US8520027B2|2010-05-14|2013-08-27|Intuitive Surgical Operations, Inc.|Method and system of see-through console overlay|
    US9858475B2|2010-05-14|2018-01-02|Intuitive Surgical Operations, Inc.|Method and system of hand segmentation and overlay using depth data|
    US8625897B2|2010-05-28|2014-01-07|Microsoft Corporation|Foreground and background image segmentation|
    US9008457B2|2010-05-31|2015-04-14|Pesonify, Inc.|Systems and methods for illumination correction of an image|
    CN101882314B|2010-07-20|2012-06-20|上海交通大学|Infrared small target detection method based on overcomplete sparse representation|
    US8649592B2|2010-08-30|2014-02-11|University Of Illinois At Urbana-Champaign|System for background subtraction with 3D camera|
    ES2395102B1|2010-10-01|2013-10-18|Telefónica, S.A.|METHOD AND SYSTEM FOR CLOSE-UP SEGMENTATION OF REAL-TIME IMAGES|
    US9247238B2|2011-01-31|2016-01-26|Microsoft Technology Licensing, Llc|Reducing interference between multiple infra-red depth cameras|
    GB2490872B|2011-05-09|2015-07-29|Toshiba Res Europ Ltd|Methods and systems for capturing 3d surface geometry|
    US8823745B2|2011-06-02|2014-09-02|Yoostar Entertainment Group, Inc.|Image processing based on depth information and color data of a scene|
    US8824797B2|2011-10-03|2014-09-02|Xerox Corporation|Graph-based segmentation integrating visible and NIR information|
    US20130095920A1|2011-10-13|2013-04-18|Microsoft Corporation|Generating free viewpoint video using stereo imaging|
    US9098908B2|2011-10-21|2015-08-04|Microsoft Technology Licensing, Llc|Generating a depth map|
    US9117281B2|2011-11-02|2015-08-25|Microsoft Corporation|Surface segmentation from RGB and depth images|
    US20130177296A1|2011-11-15|2013-07-11|Kevin A. Geisner|Generating metadata for user experiences|
    US9171393B2|2011-12-07|2015-10-27|Microsoft Technology Licensing, Llc|Three-dimensional texture reprojection|
    US9043186B2|2011-12-08|2015-05-26|Microsoft Technology Licensing, Llc|Surface normal computation on noisy sample of points|
    US8971612B2|2011-12-15|2015-03-03|Microsoft Corporation|Learning image processing tasks from scene reconstructions|
    US20130321564A1|2012-05-31|2013-12-05|Microsoft Corporation|Perspective-correct communication window with motion parallax|
    US9846960B2|2012-05-31|2017-12-19|Microsoft Technology Licensing, Llc|Automated camera array calibration|
    CN102799856A|2012-06-15|2012-11-28|天津大学|Human action recognition method based on two-channel infrared information fusion|
    CN102750533A|2012-07-05|2012-10-24|重庆大学|Infrared small and weak target detection method based on morphological component sparse representation|
    US8831285B2|2012-07-26|2014-09-09|Hewlett-Packard Development Company, L.P.|Detecting objects with a depth sensor|
    CN103279987B|2013-06-18|2016-05-18|厦门理工学院|Object quick three-dimensional modeling method based on Kinect|US9628722B2|2010-03-30|2017-04-18|Personify, Inc.|Systems and methods for embedding a foreground video into a background feed based on a control input|
    US8649592B2|2010-08-30|2014-02-11|University Of Illinois At Urbana-Champaign|System for background subtraction with 3D camera|
    US9213781B1|2012-09-19|2015-12-15|Placemeter LLC|System and method for processing image data|
    US20140307055A1|2013-04-15|2014-10-16|Microsoft Corporation|Intensity-modulated light pattern for active stereo|
    CN104427291B|2013-08-19|2018-09-28|华为技术有限公司|A kind of image processing method and equipment|
    US20150186341A1|2013-12-26|2015-07-02|Joao Redol|Automated unobtrusive scene sensitive information dynamic insertion into web-page image|
    US10469827B2|2013-12-27|2019-11-05|Sony Corporation|Image processing device and image processing method|
    US9414016B2|2013-12-31|2016-08-09|Personify, Inc.|System and methods for persona identification using combined probability maps|
    US9386303B2|2013-12-31|2016-07-05|Personify, Inc.|Transmitting video and sharing content via a network using multiple encoding techniques|
    US9485433B2|2013-12-31|2016-11-01|Personify, Inc.|Systems and methods for iterative adjustment of video-capture settings based on identified persona|
    US9589359B2|2014-04-24|2017-03-07|Intel Corporation|Structured stereo|
    US10432896B2|2014-05-30|2019-10-01|Placemeter Inc.|System and method for activity monitoring using video data|
    US10043078B2|2015-04-21|2018-08-07|Placemeter LLC|Virtual turnstile system and method|
    US10380431B2|2015-06-01|2019-08-13|Placemeter LLC|Systems and methods for processing video streams|
    CN105469447A|2014-09-11|2016-04-06|富泰华工业（深圳）有限公司|Point-cloud boundary right-angle side repairing system and method|
    DE102014115292A1|2014-10-21|2016-04-21|Connaught Electronics Ltd.|Method for providing image files from a camera system, camera system and motor vehicle|
    US10321100B2|2014-12-02|2019-06-11|Ademco Inc.|System and method of foreground extraction for digital cameras|
    JP6642970B2|2015-03-05|2020-02-12|キヤノン株式会社|Attention area detection device, attention area detection method, and program|
    US10600169B2|2015-03-26|2020-03-24|Sony Corporation|Image processing system and image processing method|
    US9563962B2|2015-05-19|2017-02-07|Personify, Inc.|Methods and systems for assigning pixels distance-cost values using a flood fill technique|
    US9916668B2|2015-05-19|2018-03-13|Personify, Inc.|Methods and systems for identifying background in video data using geometric primitives|
    US10580154B2|2015-05-21|2020-03-03|Koninklijke Philips N.V.|Method and apparatus for determining a depth map for an image|
    WO2018005513A1|2016-06-28|2018-01-04|Foresite Healthcare, Llc|Systems and methods for use in detecting falls utilizing thermal sensing|
    US9607397B2|2015-09-01|2017-03-28|Personify, Inc.|Methods and systems for generating a user-hair-color model|
    CN105827998A|2016-04-14|2016-08-03|广州市英途信息技术有限公司|Image matting system and image matting method|
    JP6754610B2|2016-05-18|2020-09-16|株式会社デンソーアイティーラボラトリ|Arithmetic processing unit, arithmetic processing method, and program|
    DK3466068T3|2016-06-03|2021-02-15|Utku Buyuksahin|A SYSTEM AND A METHOD FOR TAKING AND GENERATING 3D IMAGES|
    US9883155B2|2016-06-14|2018-01-30|Personify, Inc.|Methods and systems for combining foreground video and background video using chromatic matching|
    CN106296728B|2016-07-27|2019-05-14|昆明理工大学|A kind of Segmentation of Moving Object method in the unrestricted scene based on full convolutional network|
    CN106200249A|2016-08-30|2016-12-07|辽宁中蓝电子科技有限公司|Structure light and RGB sensor module monoblock type integrated system 3D camera|
    US10701244B2|2016-09-30|2020-06-30|Microsoft Technology Licensing, Llc|Recolorization of infrared image streams|
    US9881207B1|2016-10-25|2018-01-30|Personify, Inc.|Methods and systems for real-time user extraction using deep learning networks|
    WO2018129104A1|2017-01-03|2018-07-12|Owlii Inc.|Processing holographic videos|
    JP6814053B2|2017-01-19|2021-01-13|株式会社日立エルジーデータストレージ|Object position detector|
    WO2018191648A1|2017-04-14|2018-10-18|Yang Liu|System and apparatus for co-registration and correlation between multi-modal imagery and method for same|
    CN107169995B|2017-05-05|2020-03-10|武汉理工大学|Self-adaptive moving target visual detection method|
    CN108961316A|2017-05-23|2018-12-07|华为技术有限公司|Image processing method, device and server|
    US10586383B2|2017-06-20|2020-03-10|Microsoft Technology Licensing, Llc|Three-dimensional object scan using data from infrared sensor|
    US10674096B2|2017-09-22|2020-06-02|Feedback, LLC|Near-infrared video compositing|
    US10270986B2|2017-09-22|2019-04-23|Feedback, LLC|Near-infrared video compositing|
    US10560645B2|2017-09-22|2020-02-11|Feedback, LLC|Immersive video environment using near-infrared video compositing|
    US10612912B1|2017-10-31|2020-04-07|Facebook Technologies, Llc|Tileable structured light projection system|
    CN109889799B|2017-12-06|2020-08-25|西安交通大学|Monocular structure light depth perception method and device based on RGBIR camera|
    US11004207B2|2017-12-06|2021-05-11|Blueprint Reality Inc.|Multi-modal data fusion for scene segmentation|
    US10783668B2|2017-12-22|2020-09-22|Samsung Electronics Co., Ltd.|Handling duplicate points in point cloud compression|
    CN111727437A|2018-01-08|2020-09-29|远见汽车有限公司|Multispectral system providing pre-crash warning|
    US10803596B2|2018-01-29|2020-10-13|HypeVR|Fully automated alpha matting for virtual reality systems|
    CN108537814B|2018-03-14|2019-09-03|浙江大学|A kind of three-dimensional sonar point cloud chart based on ViBe is as dividing method|
    US10521926B1|2018-03-21|2019-12-31|Facebook Technologies, Llc|Tileable non-planar structured light patterns for wide field-of-view depth sensing|
    CN108648225A|2018-03-31|2018-10-12|深圳奥比中光科技有限公司|Target image obtains System and method for|
    CN108564613A|2018-04-12|2018-09-21|维沃移动通信有限公司|A kind of depth data acquisition methods and mobile terminal|
    US10878577B2|2018-12-14|2020-12-29|Canon Kabushiki Kaisha|Method, system and apparatus for segmenting an image of a scene|
    JP2020204856A|2019-06-17|2020-12-24|株式会社バンダイナムコアミューズメント|Image generation system and program|
    WO2021058841A1|2019-09-27|2021-04-01|Sigma Technologies, S.L.|Method for the unsupervised measurement of the dimensions of an object using a view obtained with a single camera|
    US11076111B1|2019-11-13|2021-07-27|Twitch Interactive, Inc.|Smart color-based background replacement|
    US20210366096A1|2020-05-22|2021-11-25|Robert Bosch Gmbh|Hazard detection ensemble architecture system and method|
    CN113066115A|2021-04-28|2021-07-02|北京的卢深视科技有限公司|Deep prediction network training method, device, server and readable storage medium|
    法律状态:
    2018-03-27| B15K| Others concerning applications: alteration of classification|Ipc: G06T 7/00 (2017.01), G06K 9/62 (2006.01), G06K 9/3 |
    2018-11-13| B06F| Objections, documents and/or translations needed after an examination request according [chapter 6.6 patent gazette]|
    2020-03-17| B06U| Preliminary requirement: requests with searches performed by other patent offices: procedure suspended [chapter 6.21 patent gazette]|
    2021-07-13| B350| Update of information on the portal [chapter 15.35 patent gazette]|
    2021-11-16| B09A| Decision: intention to grant [chapter 9.1 patent gazette]|
    2022-01-25| B16A| Patent or certificate of addition of invention granted [chapter 16.1 patent gazette]|Free format text: PRAZO DE VALIDADE: 20 (VINTE) ANOS CONTADOS A PARTIR DE 14/04/2014, OBSERVADAS AS CONDICOES LEGAIS. |
    优先权:
    申请号 | 申请日 | 专利标题
    US201361812233P| true| 2013-04-15|2013-04-15|
    US61/812,233|2013-04-15|
    US13/913,454|2013-06-09|
    US13/913,454|US9191643B2|2013-04-15|2013-06-09|Mixing infrared and color component data point clouds|
    PCT/US2014/033918|WO2014172230A1|2013-04-15|2014-04-14|Mixing infrared and color component data point clouds|
    [返回顶部]